On most computing devices, the user provides inputs with a keyboard and with a conventional pointing device, such as a mouse, trackball, touchpad, or other pointing button. Such input devices are much less intuitive than touch screens that enable a user to touch a spot on a displayed image and provide an input related to the object, or move the virtual object that is being touched to different positions on the display. However, touch screens are somewhat limited in the scope of user input and interaction that they can provide. For example, most touch screens are only able to detect a single point that is being touched on the screen at a time and are only responsive to a touch on the screen, which precludes more complex simultaneous multiple inputs and a greater degree of user interaction with the computing device.
A further evolution of techniques to facilitate human-machine interaction exploits computer vision technology. For example, the MIT Media Lab, as reported by Brygg Ullmer and Hiroshi Ishii in “The metaDESK: Models and Prototypes for Tangible User Interfaces,” Proceedings of UIST 10/1997:14-17,” has developed another form of “keyboardless” machine interface. The metaDESK includes a generally planar graphical surface that not only displays computing system output, but also receives user input by recognizing pattern-encoded objects placed against the graphical surface. The graphical surface of the metaDESK is facilitated using infrared (IR) lamps, an IR camera, a video camera, a video projector, and mirrors disposed beneath the surface of the metaDESK. The mirrors reflect the graphical image projected by the projector onto the underside of the graphical display surface to provide images that are visible to a user from above the graphical display surface. The IR camera can detect IR reflections from the undersurface of the objects placed on the graphical surface.
Similarly, papers published by Jun Rekimoto of the Sony Computer Science Laboratory, Inc., and associates describe a “HoloWall” and a “HoloTable” that use IR light to detect objects. It should be noted that the papers describing the “HoloWall” and “HoloTable” generally are silent regarding the details and process used for detecting objects based upon the IR light reflected from the object.
Examples of other methods and systems for detecting and responding to objects placed on or near a graphical surface are used in inventions owned by the assignee of the present application. These inventions are described, for example, in co-pending U.S. patent applications, including Ser. No. 10/813,855, entitled “Template Matching On Interactive Surface;” Ser. No. 10/814,577, entitled “Identification Of Object On Interactive Display Surface By Identifying Coded Pattern;” and Ser. No. 10/814,761 entitled “Determining Connectedness And Offset Of 3D Objects Relative To An Interactive Surface,” all of which were filed on Mar. 31, 2004.
Inventions such as the metaDESK and the other inventions noted above respond to objects on the graphical display surface. However, it would be desirable to enable a graphical display surface to respond to an input that does not involve placing a physical object on or immediately adjacent to the graphical display surface. The ability to provide such an input would enable a user to interact with the graphical display surface from a greater distance. Ideally, the interaction should encompass a variety of different types of input. For example, the user should be able to select an option that is graphically or textually displayed on the surface, or move a virtual object over the graphical display surface, or provide a graphical input that is interpreted as a stylus input or as an input by a drawing implement for producing graphic images, or interact in other ways with the graphical display surface. The ability to provide such input to the interactive display system should thus greatly enhance the user experience and expand the functionality of the system.